-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[smart_holder] Bake smart_holder functionality into class_
and type_caster_base
#5257
Conversation
Commands used: ``` git checkout bakein git diff smart_holder > ~/zd git checkout smart_holder git checkout -b bakein_sh patch -p 1 < ~/zd git checkout smart_holder \ MANIFEST.in \ README.rst \ README_smart_holder.rst \ docs/advanced/smart_ptrs.rst \ ubench/holder_comparison.cpp \ ubench/holder_comparison.py \ ubench/holder_comparison_extract_sheet_data.py \ ubench/number_bucket.h \ ubench/python/number_bucket.clif git add -A ```
std::shared_ptr<void> vptr; | ||
PYBIND11_SMART_HOLDER_PADDING(3); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
padding between bool bitfields? Why use bitfields in any case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
…YBIND11_SMART_HOLDER_PADDING`, which was meant for stress-testing only).
include/pybind11/pybind11.h
Outdated
is_method(hdl)); | ||
detail::type_info *tinfo = detail::get_type_info(typeid(T), /*throw_if_missing=*/true); | ||
if (tinfo->holder_enum_v == detail::holder_enum_t::smart_holder) { | ||
return cpp_function([pm](T &c, D value) { c.*pm = std::forward<D>(value); }, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
std::move
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done: 99b6572
Thanks for catching this!
include/pybind11/pybind11.h
Outdated
PYBIND11_NAMESPACE_END(detail) | ||
|
||
#ifdef PYBIND11_HAVE_INTERNALS_WITH_SMART_HOLDER_SUPPORT | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You could do this here:
template <typename T, typename D>
struct property_cpp_function__version6_pointer_something {
template <typename PM, detail::must_be_member_function_pointer<PM> = 0>
static cpp_function readonly(PM pm, const handle &hdl) {
detail::type_info *tinfo = detail::get_type_info(typeid(T), /*throw_if_missing=*/true);
if (tinfo->holder_enum_v == detail::holder_enum_t::smart_holder) {
return cpp_function(
[pm](handle c_hdl) -> std::shared_ptr<drp> {
std::shared_ptr<T> c_sp = detail::type_caster<
std::shared_ptr<T>>::shared_ptr_with_responsible_parent(c_hdl);
D ptr = (*c_sp).*pm;
return std::shared_ptr<drp>(c_sp, ptr);
},
is_method(hdl));
}
return property_cpp_function<T, D, void>::readonly(std::move(pm), hdl);
}
template <typename PM, detail::must_be_member_function_pointer<PM> = 0>
static cpp_function read(PM pm, const handle &hdl) {
return readonly(pm, hdl);
}
template <typename PM, detail::must_be_member_function_pointer<PM> = 0>
static cpp_function write(PM pm, const handle &hdl) {
detail::type_info *tinfo = detail::get_type_info(typeid(T), /*throw_if_missing=*/true);
if (tinfo->holder_enum_v == detail::holder_enum_t::smart_holder) {
return cpp_function([pm](T &c, D value) { c.*pm = std::move<D>(value); },
is_method(hdl));
}
return property_cpp_function<T, D, void>::write(std::move(pm), hdl);
}
};
template <typename T, typename D>
struct property_cpp_function<T,
D,
detail::enable_if_t<detail::all_of<
detail::none_of<std::is_pointer<D>,
std::is_array<D>,
detail::is_instantiation<std::unique_ptr, D>,
detail::is_instantiation<std::shared_ptr, D>>,
detail::both_t_and_d_use_type_caster_base<T, D>>::value>>
: public property_cpp_function__version6_pointer_something {};
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
|
||
#ifdef PYBIND11_HAVE_INTERNALS_WITH_SMART_HOLDER_SUPPORT | ||
|
||
# include "detail/common.h" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do these need to be ifdef guarded?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a choice (for the most part at least).
I did it that way so that it is as obvious as possible that pybind11 behaves almost exactly as before when PYBIND11_HAVE_INTERNALS_WITH_SMART_HOLDER_SUPPORT
is not defined.
(The few tiny exceptions are a compromise, #ifdef
clutter vs. tiny pieces from the smart_holder code leaking in.)
I prefer to keep this as-is, unless you believe it's better to not have conditional #include
s.
return_value_policy policy, | ||
handle parent, | ||
const std::pair<const void *, const type_info *> &st) { | ||
switch (policy) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why use a switch here? To catch policy enum changes, or what? Why not use a switch above?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To catch policy enum changes
Yes, that exactly was the idea (similar to this existing code on master):
pybind11/include/pybind11/detail/type_caster_base.h
Lines 506 to 566 in 8e7307f
switch (policy) { | |
case return_value_policy::automatic: | |
case return_value_policy::take_ownership: | |
valueptr = src; | |
wrapper->owned = true; | |
break; | |
case return_value_policy::automatic_reference: | |
case return_value_policy::reference: | |
valueptr = src; | |
wrapper->owned = false; | |
break; | |
case return_value_policy::copy: | |
if (copy_constructor) { | |
valueptr = copy_constructor(src); | |
} else { | |
#if defined(PYBIND11_DETAILED_ERROR_MESSAGES) | |
std::string type_name(tinfo->cpptype->name()); | |
detail::clean_type_id(type_name); | |
throw cast_error("return_value_policy = copy, but type " + type_name | |
+ " is non-copyable!"); | |
#else | |
throw cast_error("return_value_policy = copy, but type is " | |
"non-copyable! (#define PYBIND11_DETAILED_ERROR_MESSAGES or " | |
"compile in debug mode for details)"); | |
#endif | |
} | |
wrapper->owned = true; | |
break; | |
case return_value_policy::move: | |
if (move_constructor) { | |
valueptr = move_constructor(src); | |
} else if (copy_constructor) { | |
valueptr = copy_constructor(src); | |
} else { | |
#if defined(PYBIND11_DETAILED_ERROR_MESSAGES) | |
std::string type_name(tinfo->cpptype->name()); | |
detail::clean_type_id(type_name); | |
throw cast_error("return_value_policy = move, but type " + type_name | |
+ " is neither movable nor copyable!"); | |
#else | |
throw cast_error("return_value_policy = move, but type is neither " | |
"movable nor copyable! " | |
"(#define PYBIND11_DETAILED_ERROR_MESSAGES or compile in " | |
"debug mode for details)"); | |
#endif | |
} | |
wrapper->owned = true; | |
break; | |
case return_value_policy::reference_internal: | |
valueptr = src; | |
wrapper->owned = false; | |
keep_alive_impl(inst, parent); | |
break; | |
default: | |
throw cast_error("unhandled return_value_policy: should not happen!"); | |
} |
Ideally we'd have more like comprehensive switch
like that, to guard against accidents and misunderstandings. (I made a partial pass in the past, but the gain isn't that big, and there were always bigger fish to fry...)
return_value_policy policy, | ||
handle parent, | ||
const std::pair<const void *, const type_info *> &st) { | ||
if (policy != return_value_policy::automatic |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Positive tests are easier to follow. This function is only valid for return_value_policy::copy
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done: cb6a85f
The list of conditions started small and grew over time (to keep existing client code happy). TBH I didn't fully realize that only copy
is invalid now.
return std::shared_ptr<T>(raw_ptr, shared_ptr_parent_life_support(parent.ptr())); | ||
} | ||
|
||
std::shared_ptr<T> loaded_as_shared_ptr(void *void_raw_ptr, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit. naming. loaded_as vs. load_as?
…lining the implementations.
…parison.cpp (holder_comparison.py is NOT changed accordingly in this commit, i.e. can still only be run if the smart_holder functionality is available).
…_ptr`) as suggested by @laramiel
…s that the latest implementation of `smart_holder_from_unique_ptr()` accepts all existing `return_value_policy` enum values except `copy`.
…specializations.
Leave only two pairs of SMART_HOLDER_BAKEIN_FOLLOW_ON comments: refactoring of copyable_holder_caster, move_only_holder_caster. This is best left until after the smart_holder branch is merged into the master branch.
class_
and type_caster_base
include/pybind11/numpy.h
Outdated
@@ -1986,7 +1990,7 @@ struct vectorize_helper { | |||
// Pointers to values the function was called with; the vectorized ones set here will start | |||
// out as array_t<T> pointers, but they will be changed them to T pointers before we make | |||
// call the wrapped function. Non-vectorized pointers are left as-is. | |||
std::array<void *, N> params{{&args...}}; | |||
std::array<void *, N> params{{reinterpret_cast<void *>(&args)...}}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems like most uses of reinterpret_cast<void*>(&...)
in this cl should be changed to static_cast<void*>(&...)
instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, sorry, these came in through PR #5272 yesterday, which was merged already on master. — I'm not using rebase
and push --force
here (or anywhere really) because that invalidates a lot of commit hashes, and then I have no way to easily refer to commits from comments.
The best way to review here is to start with the "Files changed" tab near the top of this PR.
While working on PR #5272, I tried static_cast<void *>
and it built successfully, but clang-tidy (clang version 18) still complained. reinterpret_cast
was to only way to make it happy. — I didn't try to understand exactly why. — I was thinking the reinterpret_cast
s are better than // NOLINT
s.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The best way to review here is to start with the "Files changed" tab near the top of this PR.
Oh, sorry, I just now realized it doesn't look the way I was expecting. Ugh!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is a helper PR showing the net diff (what I was wrongly hoping "Files changed" would show):
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok. So perhaps make this change at head. :)
This looks reasonable to me and good to merge. |
Thanks a ton for the review @laramiel! |
…ializations. The * `copyable_holder_caster_shared_ptr_with_smart_holder_support_enabled` and * `move_only_holder_caster_unique_ptr_with_smart_holder_support_enabled` SFINAE helpers were introduced with pybind/pybind11#5257. They need to be specialized here (`std::false_type`) because native_proto_caster.h has its own specializations for * `copyable_holder_caster<ProtoType, std::shared_ptr<ProtoType>>` and * `move_only_holder_caster<ProtoType, std::unique_ptr<ProtoType>>`. PiperOrigin-RevId: 658807442
…ializations. The * `copyable_holder_caster_shared_ptr_with_smart_holder_support_enabled` and * `move_only_holder_caster_unique_ptr_with_smart_holder_support_enabled` SFINAE helpers were introduced with pybind/pybind11#5257. They need to be specialized here (`std::false_type`) because native_proto_caster.h has its own specializations for * `copyable_holder_caster<ProtoType, std::shared_ptr<ProtoType>>` and * `move_only_holder_caster<ProtoType, std::unique_ptr<ProtoType>>`. PiperOrigin-RevId: 658807442
…ializations. The * `copyable_holder_caster_shared_ptr_with_smart_holder_support_enabled` and * `move_only_holder_caster_unique_ptr_with_smart_holder_support_enabled` SFINAE helpers were introduced with pybind/pybind11#5257. They need to be specialized here (`std::false_type`) because native_proto_caster.h has its own specializations for * `copyable_holder_caster<ProtoType, std::shared_ptr<ProtoType>>` and * `move_only_holder_caster<ProtoType, std::unique_ptr<ProtoType>>`. PiperOrigin-RevId: 658807442
…ializations. The * `copyable_holder_caster_shared_ptr_with_smart_holder_support_enabled` and * `move_only_holder_caster_unique_ptr_with_smart_holder_support_enabled` SFINAE helpers were introduced with pybind/pybind11#5257. They need to be specialized here (`std::false_type`) because native_proto_caster.h has its own specializations for * `copyable_holder_caster<ProtoType, std::shared_ptr<ProtoType>>` and * `move_only_holder_caster<ProtoType, std::unique_ptr<ProtoType>>`. PiperOrigin-RevId: 658807442
…ializations. The * `copyable_holder_caster_shared_ptr_with_smart_holder_support_enabled` and * `move_only_holder_caster_unique_ptr_with_smart_holder_support_enabled` SFINAE helpers were introduced with pybind/pybind11#5257. They need to be specialized here (`std::false_type`) because native_proto_caster.h has its own specializations for * `copyable_holder_caster<ProtoType, std::shared_ptr<ProtoType>>` and * `move_only_holder_caster<ProtoType, std::unique_ptr<ProtoType>>`. PiperOrigin-RevId: 658867114
Description
This PR is a continuation of PR #5213 on top of the smart_holder branch.
Overarching goal: Prepare the smart_holder branch for merging into master, to start the pybind11 v3 line.
Note: This PR restores the existing unit tests on the master branch, with very small exceptions. See PR #5213 Description for details.
High-level trade-off:
smart_holder_type_casters
(smart_holder branch before this PR) were an add-on and thePYBIND11_INTERNALS_VERSION
did not have to be changed.For the baked-in
smart_holder
support (smart_holder branch after this PR), thePYBIND11_INTERNALS_VERSION
needs to be changed, but there is far fewer duplicated code.Major benefit: The
PYBIND11_SMART_HOLDER_TYPE_CASTERS
macro is obsolete (it still exists for backward compatibility but is a no-op).Added benefit: Testing with smart_holder as the default holder is far less important. Testing with smart_holder as the default holder will be cut back (in a follow-on PR) to one each for Linux, macOS, Windows.
NOTE: The baked-in smart_holder functionality can easily be disabled by overriding the
PYBIND11_INTERNALS_VERSION
: setting it to5
(or even4
) almost completely#ifdef
s out all code added in this PR.Work for follow-on PRs: Resolve all
SMART_HOLDER_BAKEIN_FOLLOW_ON
. This work is intentionally deferred because it will change existing code on the master branch in non-trivial ways.Suggested order for code review: (internal CL)
enum class holder_enum_t
.copyable_holder_caster
andmove_only_holder_caster
specializations forstd::shared_ptr
andstd::unique_ptr
.smart_holder_from_unique_ptr
andsmart_holder_from_shared_ptr
implementations (based on code originally in smart_holder_type_casters.h).init_instance()
specialization (based on code originally in smart_holder_type_casters.h) and modifiedproperty_cpp_function
specializations.Before reviewing the test changes, please read the description of PR #5213. Most importantly, almost all changes to tests on the master branch are undone. The test_class_sh_*.cpp,py changes in this PR are essentially pure boilerplate changes or trivial.
Suggested changelog entry: